Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. On the one hand, ChatGPT (or LLMs) serves as a general interface that provides a broad and diverse understanding of a wide range of topics. On the other hand, Foundation Models serve as domain experts by providing deep knowledge in specific domains. By leveraging both general and deep knowledge, we aim at building an AI that is capable of handling various tasks. For help or issues using the Visual ChatGPT, please submit a GitHub issue.

However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. Visual ChatGPT, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple #AI models with multi-steps. A series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models.

One of the main weak points of conversational artificial intelligence ChatGPTChatGPT is that it is limited to text only. To solve this problem, researchers at MicrosoftMicrosoft have just released a new version of ChatGPT called Visual ChatGPT. In the associated articlethey explain how they managed to integrate image support into ChatGPT without touching the AI itself. Rather than completely rebuilding ChatGPT to support different modalities (audio, images, videos…), they decided to rely on pre-existing Visual Foundation Models (VFMs), like Stable Diffusion, BLIP, Transformers, Maskformer and ControlNet. The central module of Visual ChatGPT is the request handler (Prompt Manager).

After $10Billion ChatGPT deal, now Microsoft Research recently launched new module called "Visual ChatGPT" which allows users to send their image requests via chat and receive it with editing functionality. Still we have to wait and see how smart comparing to Dall-E2. In official statement MS research says Visual ChatGPT uses different visual foundation models to let users to get the best output images. What Visual ChatGPT will do? It's very simple, If you upload an Matt black photoframe and request ChatGPT to change the colour as Deep purple and add moon object inside the frame then V-ChatGPT will do the work for you.